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SUMMARY 



A reliability growth model is an analytical model that accounts for 
changes in reliability due to design changes and other corrective actions 
taken during the development and testing phases of a reliability program. 
This paper describes the results of a Monte Carlo study comparing the 
performance characteristics of four reliability growth models that have 
been proposed in the reliability literature. 
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1. INTRODUCTION 



A reliability growth model is an analytical model that accounts for 
changes in reliability (usually an improvement) due to design changes and 
other corrective actions taken during the development and testing phases 
of a reliability program. A good reliability growth model would be use- 
ful in determining a test plan that will lead to the development of an 
acceptable product within a reasonable time period. Once a test plan is 
developed the model may be used to monitor the progress of the reliability 
program and if warranted, initiate changes such as accelerated testing. 

At any point in time during the test phase an idea as to how much addi- 
tional testing would be required to achieve a reliability goal may be 
obtained using the model. 

There are several reliability growth models available in the liter- 
ature. Some of these models are probabilistic in nature i.e., the models 
do not allow for incorporation of information obtained during the test 
phase. The second type of models, generally known as statistical models, 
involve unknown parameters that can be estimated using available test data. 
This paper discribes the results of a study comparing the performance for 
four statistical models. Two of these models are intended for use with time 
to failure data. The other two models use bemoulli type data; a "success” 
is taken to mean "the satisfactory operation for a specified period of time" 
and the specified time period is often referred to as "mission time". 



2 



A description of the reliability growth models selected for this study 



is given in Section 2. The Monte Carlo procedure used is described in 
Section 3 and Section 4 contains the methods of estimating the parameters 
of the different models. The results of the study are found in Section 5. 
Corcoran and Read [3] conducted a Monte Carlo study to compare several 
reliability growth models that includes two of the models considered in 
this paper. However, there are several differences in the methodology used 
in the two studies. 



2. Description of the Reliability Growth Models . 



Model I: Duane [4] developed this model based on his analysis of 

large quantities of data on various types of airborne equip- 
ment. If T is the accumulated test time on a given piece 
of equipment, and is the instantaneous failure rate at 

time T then, Duane's model states that 

= B(l-a)T"“ 

where g and a are unknown parameters. 

Model II: Suppose that (i) items are put on test sequentially and after 

each failure an appropriate corrective action is taken and 
(ii) the time to failure on the i^^ test has an exponential 
distribution with parameter . Then, Weiss [5] assumes 

that the parameters satisfy the equation 



X. 

1 



B 4- i 
Ai 



for some unknown parameters A and B . 
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Model III: Let 



p == P [’’mission success” after the failure has been 

r 

observed and appropriate corrective action has been 
taken] 



Chernoff and Woods [ 2 ] model p^ as an increasing function 
of r defined by 



^r+l 



1 _ e-(“+er) 



Model IV: Wolman [ 6 ] developed this model assuming that 

(i) testing is conducted sequentially and each trial results 
in a ’’mission success” or a ’’mission failure” 

(ii) on each trial one of two types of failures can occur; the 
first type is an inherent or non-correctable failure and 
the second type of failure is a transient failure that can 
be eliminated by corrective action and 

(iii) whenever a test results in a ’’failure” an attempt is made 
to eliminate the failure mode and the attempt may result 
in a) either a complete elimination b) a partial elimi- 
nation or c) no effect on the failure mode. 

Let 

r = probability of an inherent failure, constant from 
test to test 

q = probability of a transient failure at the beginning 
of the test program 

= factor by which the corrective action after the i^^ 

failure, reduces the probability of a transient failure 
i-1 

a. = IT g the cumulative factor by which the initial 

i j-oJ 

probability of a transient failure is reduced as a 
result of the corrective actions after each of the 
i failures and 
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= probability of a "mission success" on the 
(n+l)st test if i failures are observed 
in the first n tests 



The Wolman model assumes that 






1 - r - qa. 

1 



Wolman points out that in many situations a good approxi- 
mation is obtained by assximing that the improvement factors 
are all equal to a constant value 6 . In this case, 
the model equation reduces to 

■ 1 - r - 

The latter version with g = ,2 , ,5 and .8 is used in 
this study. 



3. Monte Carlo Procedure 

In order to compare the growth models described in the previous section, 
five decreasing sequences of X (failure rate) values are selected. Each 
sequence consists of 16 values ranging between .70 and .05; this represents 
a growth in reliability, assuming that all times are measured in mission 
units, from an initial value of .50 to a final value of .95 approximately. 

The growth patterns represented in these sequences vary from a very rapid 
growth to a very slow growth. 

For each of the 16 X's in a given sequence 10 exponential failure times 
^ii • • • > > j=l,2,...,10 are generated. Since a reliability growth model is 



5 



to predict the X values that are in practical situations unknown, the 
generated data is used to first estimate the X values in a given sequence 
in two ways. In the first method the j * ^ treated as the observed 

exponential failure times and in the second case the j * ® converted 

into bernoulli data by noting whether each t^^ excedes 1 or not. This is 
done to study the effect of different data types on the prediction capa- 
bilities of the growth models. The estimated X values are used to em- 
pirically determine the unknown parameters of the growth models using 
least squares techniques. To account for statistical variability the whole 
process described above is replicated 100 times resulting in 100 empirical 
growth curves for each of the X sequences and each of the reliability 
growth models. 

If X^ , i = 1,2,..., 16 are the failure rates in a given sequence 
100 predicted values X* are obtained from the empirical curves for each 
of the models. The "goodness" of a reliability growth model is now measured 
using the following three measures: 

16 



\ - I 

i=i 






16 

I 

i=l 



M_ = max 
^ i 



^i- 


^i* 


^i 




X. - 
r 




X. 

1 


- 


1 

•H 


X? 

1 


X 



where X* is the average of the 100 predicted values X* . All three 

measures are relative measures i.e., each difference X. - X* is divided 

1 i 
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by the true value in computing the measures. The reason for choosing 

relative measures is that it would be important for a growth model to pre- 
dict the small X values at the end of a sequence rather well. In other 
words, these measures attach more weight to the differences X^ - X* at 
the end of each decreasing sequence. 



4. Estimation of Parameters 



As indicated earlier the generated failure times t^^, j = 1,2,..., 10 
are used to estimate the failure rates X^ in two different ways. For the 
case where the used directly the estimator of X^ is the maximum 



'ij 

^ — — 1 
likelihood estimator X^ = where t^ = — 



10 

I 

j=i 






Since models 



III and IV describe the growth in p^ the probability of a mission success, 
X^ is converted into an estimator of p^ by noting that 

p^ = P [mission success on i^^ trial] 



-X, 



= P[ ti^ > 1] = e 



-A, 



Thus, the maximum likelihood estimator of p^ is p^ =e 



In the second 



case, the t, ,’s are transformed to bernoulli data and the X, 
ij 1 

and p^ are estimated from the transformed data as follows: 

Let 



^ij 



1 

0 



if 




if 


'ij ^ 
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k 

and let Y. = 7 X. . . Then, Y. has a binomial distribution with 
^ j=i 

parameters p. and k . The maximum likelihood estimators of p, and 

^ ^ Y. - ^ ^ 

X. based on Y. are p. = and X. = -In p. respectively. The 

1 1X1^ X X 

methods for obtaining the empirical growth curves and the predicted X 
values from the growth curves are now described. Since these methods are 
the same for both sets of estimators X. , p. and X. > p. the symbols 
X. , p. will be used to denote either of the two sets of estimators. Also, 

X X 

the symbols X* , p* denote the X^ , p^ values as predicted by the 
empirical growth curves. 

For model I the equation describing the relationship between the 
failure rate X^ and the total test time T is 

= 3(l-a)T"“ 



or equivalently 



In X^ = ln[B(l-a)] - a In T 



The parameters a and B are estimated by replacing X with X. 

i 10 ^ ^ 

and T with T. = ^ ^ t^, in the latter equation and then apply- 

^ £=1 j=l 

ing standard regression methods. Once the least squares estimators a 
and B are obtained, the predicted values XJ of the failure rates X^ 
are given by 



X* = B(1 - a)T. 



-a 



X 1,2,. ..,16 



The parameters A and B of model II are estimated using regression 
methods on the relationship 



B 1 1 

A i A 



and the predicted values are 

B + i 



X* = 

X 



Ai 



i = 1, 2 , . . . , 16 

X 1,2,. ..,16 
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For model III the regression equation has the form 



-Zn(l - p^) = a + 3(i-l) i = 

and for model IV the equation is 

1 - = r + qg^ ^ i = 1,2,...,16 with 6 known 

The rest of the procedure is the same as for models I and II. 

5. SIMULATION RESULTS 

The results of the simulation are presented in tables 1.1 to 1.5, 
The five tables correspond to the five sets of 16 A values used in 
generating the failure times. These A values are listed at the top 
of the tables. As indicated earlier the five sets chosen represent re- 
liability growth ranging between a slow growth to a rapid growth. Also, 
the selection of the A’s in these sets are made so as to generally 
follow the growth indicated by the models that are being tested. The 
entries in the tables are the observed values of the three measures 
^1^^2*^3 each of the four models with three cases under model IV. 

It may be observed from tables 1.1 to 1.5 that with time to failure 
data model I performed better than the others in all cases except one 
(table 1,4) where model III was best. For attributes data model IV 
with 3 = .8 appears to outperform the other models. This result coin- 
cides with a similar conclusion by Bresenham [1]. 

In order to study the case where exactly one item is put on test at 
any time, the simulation was rerun generating only one failure time t^ 
instead of 10 failure times t^^ for each of the 16 A*s in a given 
sequence. There was a difference in the generation of attribute data 
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from the failure times. Let denote the largest integer less than or 

equal to t^ , the time to failure on the i^^ trial with failure rate 
X. , i = 1, 2, • . . , 16 . Since the t. are assumed to be measured in mission 
units is equivalent to the number of full missions completed before the 

occurrence of a failure. Thus, X^ has a geometric distribution with param- 
eter p^ , the probability of a mission success on the i^^ trial. The 
maximum likelihood estimator of p^ is p^ = 1/(X^ + 1). Except for this 
difference in the way the are estimated the rest of the procedure 

is the same as before. The results of this second study are in tables 2.1 
to 2.5. In this case, model III appears to do uniformly well with time to 
failure data and with attributes data model III and model IV with 6 = .8 
seems to perform better than the others. 

On the basis of this study, the following general conclusions may be 
drawn: 

(1) with time to failure data and several items on test simultaneously, 
model I appears to predict reliability growth accurately 

(2) with attributes data and several items on test model IV with 6 = .8 
performs well 

(3) with time to failure data and one item on test model III is a good 
model to use and 

(4) with attributes data and one item on test either model III or 
model IV with 6 = .8 may be used. 

Graphs comparing true growth curves (curves down through the X values 
used in the study) with the curves estimated by model I using time data and 
model IV using attribute data with 6 = .8 are presented in Figures 1.1 to 1.5. 
Figures 2.1 to 2.5 compare the true growth curves with the curves estimated by 
model III using time and attribute data and model IV using attribute data 
with 3 = .8 . 

The computer programs used in the study are attached. 
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TABLE 1,1 



True Values of Lambda: . 702 .434 .320 .255 .213 .183 .161 



14A .130 


.118 


109 . 101 


.0936 


.0876 


.0823 


.0776 


MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


0.69 


0.04 


0.10 


9.46 


6.23 


0.99 


Model II 


2.A5 


0.65 


0.40 


2.28 


0.54 


0.38 


Model III 


2.27 


0.44 


0.37 


7.91 


4.66 


.778 


Model IV 
6 = (.2) 


7.49 


5.16 


1.13 


6. 70 


4.04 


0.99 


Model IV 
6 = (.5) 


4.52 


2.13 


0.79 


4.09 


1.57 


0.67 


Model IV 
3 = (.8) 


1.50 


0.24 


0.23 


1.55 


0.18 


0.16 
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TABLE 1.2 



True Values of Lambda: .700 .573 .508 .466 .435 .411 .392 

.376 .362 .350 .340 .330 .322 .314 .307 .301 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


0.68 


0.04 


0.11 


2.69 


0.50 


0.22 


Model II 


1.67 


0.27 


0.23 


1.35 


0.18 


0.20 


Model III 


1.04 


0.09 


0.15 


2.76 


0.58 


0.32 


Model IV 
6 = (.2) 


2.67 


0.58 


0.33 


2.21 


0.40 


0.25 


Model IV 
6 = (.5) 


1.85 


0.32 


0.27 


1.58 


0.20 


0.18 


Model IV 
6 = (.8) 


1.09 


0.08 


0.11 


0.39 


0.01 


0.06 
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TABLE 1.3 



True Values of Lambda: . 700 

.0933 .0837 .0760 .0697 



.353 .238 .180 .145 .122 .106 

.0644 .0600 .0562 .0529 .0500 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


0. 78 


0.06 


0.11 


11.00 


8.04 


0.82 


Model II 


1.61 


0.16 


0.11 


0.92 


0.06 


0.08 


Model III 


3.17 


0.86 


0.50 


10.20 


6.94 


0.86 


Model IV 
6 = (.2) 


8.42 


6.57 


1.26 


7.33 


4.88 


1.08 


Model IV 
8 = (.5) 


4.22 


1.93 


0.76 


3.59 


1.28 


0.61 


Model IV 
8 = (.8) 


3.87 


1.16 


0.43 


3.81 


1.13 


0.44 
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TABLE 1.4 



True -Values of Lambda: . 700 .564 .460 .379 

.186 .157 .132 .112 .0953 .0810 .0689 



.315 .263 .221 

.0587 .0500 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


1.88 


0.48 


0.55 


11.60 


13.80 


2.91 


Model II 


8.32 


8.93 


1.85 


8.01 


8.16 


1.78 


Model III 


0.56 


0.02 


0.06 


11.90 


19.80 


3.85 


Model IV 
B = (.2) 


14.10 


24.30 


3.04 


12.90 


20.10 


2.77 


Model IV 
6 = (.5) 


10.60 


14.20 


2.39 


9.75 


11.40 


2.15 


Model IV 
6 = (.8) 


2.45 


0.92 


0.65 


1.95 


0.47 


0.48 



14 



TABLE 1.5 



True Values of Lambda: .700 

.0880 .0757 .0672 .0613 



.456 .315 .227 .170 .132 .106 

.0572 .0544 .0524 .0510 .0500 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


1.23 


0.13 


0.14 


11.80 


9.74 


1.54 


Model II 


3.67 


1.07 


0.38 


3.73 


1.11 


0.39 


Model III 


3.70 


1.09 


0.41 


9.88 


6.60 


0.88 


Model IV 
8 = (.2) 


11.80 


12.10 


1.47 


10.80 


10.20 


1.34 


Model IV 
8 = (.5) 


6.57 


3.93 


0.88 


6.01 


3.11 


0.78 


Model IV 
8 = (.8) 


3.68 


1.12 


0.47 


3.47 


1.07 


0.53 
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TABLE 2.1 



True Values of Lambda: .702 .434 .320 .255 .213 

.144 .130 .118 .109 .101 .0936 .0876 .0823 



.183 .161 

.0776 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


19.20 


43.10 


5.44 


5.71 


6.15 


2.21 


Model 11 


13.20 


11.30 


10.80 


30.7 


83.8 


4.83 


Model III 


9.63 


6.37 


0.76 


4.56 


1.68 


.559 


Model IV 
6 = (.2) 


40.50 


297.00 


15.40 


16.50 


85.00 


6.26 


Model IV 
6 = (.5) 


36.70 


229.00 


13.50 


13.50 


63.30 


7.74 


Model IV 
B = (.8) 


23.00 


35.00 


35.00 


3.88 


1.37 


0.57 
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TABLE 2.2 



True Values of Lambda: .700 .573 .508 .466 .435 .411 .392 

.376 .362 .350 .340 .330 .322 .314 .307 .301 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


18.60 


38.40 


5.02 


3.73 


3.51 


1.78 


Model II 


118.00 


877.00 


8.35 


8.68 


14.10 


3.15 


Model III 


6.36 


2.70 


0.49 


2.16 


.354 


.240 


Model IV 
6 = (.2) 


24.60 


169.00 


12.60 


7.80 


24.40 


4.86 


Model IV 
6 = (.5) 


24.40 


164.00 


12.40 


7.27 


23.10 


4.75 


Model IV 
3 = (.8) 


13.80 


12.70 


1.61 


3.83 


3.52 


1.76 
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TABLE 2.3 



True Values of Lambda: .700 

.0933 .0837 .0760 .0697 



353 .238 .180 .145 .122 .106 

.0644 .0600 .0562 .0529 .0500 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


19.20 


41.20 


5.20 


5.47 


6.17 


2.26 


Model II 


127.00 


1010.00 


8.68 


50.80 


216.00 


6.40 


Model III 


11.10 


8.53 


0.93 


4.79 


1.76 


0.64 


Model IV 
3 = (.2) 


45.80 


303.00 


14.60 


15.90 


60.50 


7.30 


Model IV 
3 = (.5) 


41.40 


247.00 


13.40 


10.40 


29.50 


5.12 


Model IV 
3 = (.8) 


25.90 


44.50 


2.12 


3.54 


.834 


0.29 



18 



TABLE 2.4 



True Values of Lambdas .700 .564 .460 .379 

.186 .157 .132 .112 .0953 .0810 .0689 



.315 .263 .221 

.0587 .0500 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


21.00 


70.00 


7.49 


8.13 


11.90 


2.91 


Model II 


182.00 


2670.00 


25.50 


27.00 


77.50 


5.02 


Model III 


9.91 


7.21 


12.90 


4.67 


2.46 


0.93 


Model IV 
0 = (.2) 


49.20 


433.00 


17.40 


25.90 


167.00 


11.90 


Model IV 
B = (.5) 


47.50 


475.00 


19.30 


23.30 


157.00 


11.80 


Model IV 
B = (.8) 


28.40 


81.60 


5.72 


8.54 


10.10 


1.91 
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TABLE 2.5 



True Values of Lambda: .700 

.0880 .0757 .0672 .0613 



.A56 .315 .227 .170 .132 .106 

.0572 .0544 .0524 .0510 .0500 



MODEL 


TIME DATA 


ATTRIBUTE DATA 


NUMBER 


Ml 


M2 


M3 


Ml 


M2 


M3 


Model I 


19.50 


45.10 


5.62 


6.08 


6.46 


2.27 


Model II 


145.00 


1370.00 


11.10 


48.50 


205.00 


6.34 


Model III 


10.90 


9.22 


1.11 


5.50 


2.21 


0.59 


Model IV 
3 = (.2) 


52.20 


421.00 


17.40 


25.10 


204.00 


13.90 


Model IV 
3 = (.5) 


46.30 


353.00 


16.50 


18.20 


124.00 


10.90 


Model IV 

3 = (.8) 


25.90 


45.70 


2.06 


3.42 


0.90 


0.35 
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FIGURE 1.1 GRAPH OF PREDICTED FAILURE RATES FOR 

MODEL L MODEL IU(.8), AND TRUE RATE 
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